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Art Unit 

2121 



The MAILING DATE of this communication appears on the cover sheet with the correspondence address 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE (31 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S. C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )[3 Responsive to communication(s) filed on 21 March 2001 . 
2a)D This action is FINAL. 2b)^ This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 
Disposition of Claims 

4) ^ Claim(s) 1-42 is/are pending in the application. 

4a) Of the above claim(s) 1-21 is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) KI Claim(s) 22-31. 33-38. 40-42 is/are rejected. 

7) ^ Claim(s) 32 and 39 is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) ^ The specification is objected to by the Examiner. 

10) E3 The drawing(s) filed on 21 March 2001 is/are: a)S accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

11) D The proposed drawing correction filed on is: a)D approved b)D disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§ 1 1 9 and 1 20 

13) ^ Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 

a)KI All b)D Some*c)D None of: 

1 Certified copies of the priority documents have been received. 

2. ^ Certified copies of the priority documents have been received in Application No. 09/787.698 . 

3. ^ Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

15) Q Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121. 
Attachment(s) 

1 ) ^ Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-413) Paper No(s). . 

2) [3 Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) □ Notice of Informal Patent Application (PTO-152) 

3) [3 Information Disclosure Statement(s) (PTO-1449) Paper No(s) 3. 6) □ Other: 
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Examiner's Detailed Office Action 



1. This Office Action is responsive to application 09/787,698, filed March 21, 2001. 

2. Claims 1-21 have been canceled. Claims 22-42 have been added and examined. 



3. The lengthy specification has not been checked to the extent necessary to determine the 
presence of all possible minor errors e.g., on page 4, applicant has made a reference to M. Heger, 
and spelled it as "Heger." However, on page 5, applicant has spelled it as "Hager." Applicant's 
cooperation is required in correcting any errors of which applicant may become aware in the 
specification. Appropriate correction is required. 



4. Claims 32 & 39 are objected to as being dependent upon a rejected base claim, but would 
be allowable if rewritten in independent form including all of the limitations of the base claim 
and any intervening claims. 



Specification Objection 



Claim Objection(s) 
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Claim Rejections - 35 USC § 102 



5. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(a) the invention was known or used by others in this country, or patented or described in a printed publication in this 
or a foreign country, before the invention thereof by the applicant for a patent. 

6. Claim 22 are rejected under 35 U.S.C. 102(b) as being anticipated by 

Ralph Neuneier, "Enhanced Q-Learning for Optimal Asset Allocation" Advances in Neural 
Information Processing Systems (1997). 

Regarding Claim 22: 
Neuneier teaches, 



A method for computer-aided determination of a sequence of actions for a system 
having states, the method comprising the steps of: performing a transition in state 
between two states on the basis of an action [(2.1 Q-Learning for asset alloca- 
tion, page 937-938)]; determining the sequence of actions to be performed such 
that a sequence of states results from the sequence of actions [(2.1 Q-Learning 
for asset allocation, page 937-938)]; optimizing the sequence of steps with re- 
gard to a prescribed optimization function, including a variable parameter [(2.1 
Q-Learning for asset allocation, page 937-938)]; and using the variable para- 
meter to set a risk which the resulting sequence of states has with respect to a 
prescribed state of the system. [(3.1 Risk-adjusted MDP's, page 941)] 
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Claim Rejections - 35 USC § 103 



7. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 
102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the 
subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill 
in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the 
invention was made. 

8. Claims 23-31 and 33-35 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Ralph Neuneier, "Enhanced Q-Learning for Optimal Asset Allocation" Advances in Neural 
Information Processing Systems (1997) further in view of Werbos (USPN 6,169,981). 

The Neuneier reference has been discussed above and does not explicitly teach the limitation 
of claims 23-24, 26, 29-30. However, Werbos teach the limitations of claims 23-24, 26, 29-30. 

Regarding Claim 23: 
Werbos teaches, 

using approximate dynamic programming for the purpose of determination, [(col. 7, line 37-38 
"In Approximate Dynamic Programming (ADP) ... model to approximate this function J.")\ It 
would have been obvious at the time the invention was made to a person having ordinary skill in 
the art to which said subject matters pertains, to employ approximate dynamic programming 
(ADP), because, ADP allows for general-purpose programming for doing optimization over time 
by using learning and approximation. Moreover, ADP has emerging as one of the most 
promising mathematical and computational approaches to resolve the problem of nonlinear, large 
scale dynamic control problems under uncertainty. 
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Regarding Claim 24: 
Werbos teaches, 

basing the approximate dynamic programming Q-learning. [(col. 93, line 38-50 "The basic 
equation for "J" given in ...P. Werbos, ... Still, these designs are all a step up from the 1983 
BSA design")] It would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matters pertains, to employ approximate 
dynamic programming (ADP) Q-Learning, because, Q-Learning is a stochastic approximation- 
like method that iterates on the Q-factors. 

Regarding Claim 25: 
Neuneier teaches, 

forming an optimization function within Q-learning in accordance with the following rule: . . . 
[(as per applicant's own admission (page 2-4 )] It would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matters 
pertains, to employ an optimization function for specifying the expectation and gains 
accumulated over time. 

Regarding Claim 26: 
Werbos teaches, 

basing the approximate dynamic programming on TD(X)-learning. [(col. 110, line 39-45 "In any 

event, the procedure implied ... is considerably more general than any of the original 
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formulation.")] It would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matters pertains, to employ approximate 
dynamic programming on TD(X)-learning, because, the temporal difference method is a type of 
reinforcement learning that learn through exploiting the difference in evaluating actions in 
successive steps and thus handling sequences in an incremental manner. 

Regarding Claim 27: 
Neuneier teaches, 

forming an optimization function within Q-learning in accordance with the following rule: . . . 
[(as per applicant's own admission (page 8-10 )] It would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matters 
pertains, to employ an optimization function for specifying the expectation and gains 
accumulated over time. 

Regarding Claim 28: 
Neuneier teaches, 

using a technical system to determine the sequence of actions before the determination measured 
values are measured, [(as per applicant's own admission (page 1)] It would have been obvious 
at the time the invention was made to a person having ordinary skill in the art to which said 
subject matters pertains, to predict a planning mechanism to determine the sequence of actions to 
reach the chosen goal state. 
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Regarding Claim 29: 
Werbos teaches, 

subjecting the technical system to open-loop control in accordance with the sequence of actions, 
[(col. 100, line 20-28 "In summary, the concept of learning—while not all-encompassing— could 
be expected to give us a parsimonious understanding ... such research is often useful only when 
it is integrated into a specific plan to use the sensors within a larger control loop")] It would 
have been obvious at the time the invention was made to a person having ordinary skill in the art 
to which said subject matters pertains, to employ open- loop control, because an open-loop 
control system is controlled directly, and only, by an input signal, without the benefit of 
feedback. The basic units of this system are an amplifier and a motor. The amplifier receives a 
low-level input signal and amplifies it enough to drive the motor to perform the desired job. 
Open-loop control systems are not as commonly used as closed-loop control systems because 
they are less accurate. 

Regarding Claim 30: 
Werbos teaches, 

subjecting the technical system to closed- loop control in accordance with the sequence of 
actions, [(col. 130, line 19-27 "Balakrishnan has mainly studied problems in aircraft and missile 
control ... done tests demonstrating robustness and closed-loop "re-adaptation ")\ It would 
have been obvious at the time the invention was made to a person having ordinary skill in the art 
to which said subject matters pertains, to employ closed- loop control, because process 
improvement can be obtained by means of closed-loop control i.e., with reliable. While, the basic 
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idea of feed-back control is to correct disturbances, which may appear in real production 
processes. With reliable closed-loop control the process can often be run at paths in the state 
space which promise a better process performance. 

Regarding Claim 3 1 : 
Neuneier teaches, 

modeling the system as a Markov decision problem, [(as per applicant's own admission (page 
1)] It would have been obvious at the time the invention was made to a person having ordinary 
skill in the art to which said subject matters pertains, to employ a natural approach to developing 
control architectures for complex tasks is to design multiple simple components, each addressing 
one aspect of the task domain, and provide a framework for combining the results. When an 
agent must learn these components by interacting with the environment, as in a Markov decision 
problem, the designer must take care not to sacrifice an optimal solution for a convenient 
representation. 

9. Claims 33-35 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Ralph Neuneier, "Enhanced Q-Learning for Optimal Asset Allocation" Advances in Neural 
Information Processing Systems (1997), further in view of Howard (USPN 6,336,109). 
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Regarding Claim 33: 
Howard teaches, 

using the system as a communications system, [(col. 7, line 65 to col. 8, line 5 "Another 
possibility is to use a neural network classifier ...specifically for telecommunications 
applications")] It would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matters pertains, to employ a processor is 
used in a communications system, because the processor {computer system) can be used to 
analyze data about the transmission of messages in a communications system. 

Regarding Claim 34: 
Howard teaches, 

using the system to carry out access control in a communications network, [(col. 1, line 11-16 

"In the telecommunications field, large amounts of data are available, for example about 
customer behavior and telephone usage. This data contains potentially useful information for 
many purposes such as detection of fraud, marketing, billing, maintenance planning and fault 
detection.")] It would have been obvious at the time the invention was made to a person having 
ordinary skill in the art to which said subject matters pertains, to employ a processor is used to 
carry out access control in a communications network, because the processor {computer system) 
can be used to interpret, analyze and investigate user patterns of behavior in detecting e.g., 
telecommunications fraud. 
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Regarding Claim 35: 
Howard teaches, 

using the system to carry out a routing in a communications network, [(col. 1, line 11-16 "In the 

telecommunications field, large amounts of data are available, for example about customer 
behavior and telephone usage. This data contains potentially useful information for many 
purposes such as detection of fraud, marketing, billing, maintenance planning and fault 
detection. 6 ')] It would have been obvious at the time the invention was made to a person having 
ordinary skill in the art to which said subject matters pertains, to employ a processor is used to 
carry out routing in a communications network, because the processor (computer system) can be 
used to interpret, analyze and investigate user patterns of behavior in detecting e.g., in relation to 
faults in a communications network and encryption key management. 

10. Claim 36 is rejected under 35 U.S.C. 102(b) as being anticipated by 

Walter J. Gutjahr, "Failure Risk Estimation via Markov Software Usage Models" Department of 

Statistics, Operations research and Computer Science, University of Vienna, Austria (1997). 

Regarding Claim 36: 
Gutjahr teaches, 

A system for determining a sequence of actions for a system having states, 
wherein a transition in state between two states is performed on the basis of an 
action, the system comprising: a processor for determining a sequence of actions, 
whereby a sequence of states resulting from the sequence of actions is optimized 
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with regard to a prescribed optimization function, and the optimization function 



includes a variable parameter for setting a risk which the resulting sequence of 



states has with respect to a prescribed state of the system. [(Abstract "... risk 



estimate ...) and (5. Experimental results, page 189-190, Table 1)] 



Claim Rejections - 35 USC § 103 



11. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 
102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the 
subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill 
in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the 
invention was made. 



12. Claims 37-38 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Walter J. Gutjahr, "Failure Risk Estimation via Markov Software Usage Models" Department 
of Statistics, Operations research and Computer Science, University of Vienna, Austria (1997), 
further in view of Werbos (USPN 6, 1 69,98 1 ). 

The Gutjahr reference has been discussed above and does not explicitly teach the limitation 
of claim 37-38. However, Werbos teaches the limitations of claims 37-38. 
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Regarding Claim 37: 
Werbos teaches, 

the processor is used to subject a technical system to open-loop control, [(col. 100, line 20-28 
"In summary, the concept of learning—while not all-encompassing— could be expected to give us 
a parsimonious understanding ... such research is often useful only when it is integrated into a 
specific plan to use the sensors within a larger control loop.")] It would have been obvious at the 
time the invention was made to a person having ordinary skill in the art to which said subject 
matters pertains, to employ open-loop control, because an open-loop control system is controlled 
directly, and only, by an input signal, without the benefit of feedback. The basic units of this 
system are an amplifier and a motor. The amplifier receives a low-level input signal and 
amplifies it enough to drive the motor to perform the desired job. Open-loop control systems are 
not as commonly used as closed-loop control systems because they are less accurate. 

Regarding Claim 38: 
Werbos teaches, 

the processor is used to subject a technical system to closed-loop control, [(col. 130, line 19-27 
"Balakrishnan has mainly studied problems in aircraft and missile control ... done tests 
demonstrating robustness and closed-loop "re-adaptation ")\ It would have been obvious at the 
time the invention was made to a person having ordinary skill in the art to which said subject 
matters pertains, to employ closed-loop control, because process improvement can be obtained 
by means of closed-loop control i.e., with reliable. While, the basic idea of feed-back control is 
to correct disturbances, which may appear in real production processes. With reliable closed-loop 
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control the process can often be run at paths in the state space which promise a better process 
performance. 

13. Claims 40-42 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Walter J. Gutjahr, "Failure Risk Estimation via Markov Software Usage Models" Department 
of Statistics, Operations research and Computer Science, University of Vienna, Austria (1997), 
further view of Howard (USPN 6,336,109). 

Regarding Claim 40: 
Howard teaches, 

the processor is used in a communications system, [(col. 7, line 65 to col. 8, line 5 "Another 
possibility is to use a neural network classifier .specifically for telecommunications 
applications")] It would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matters pertains, to employ a processor is 
used in a communications system, because the processor (computer system) can be used to 
analyze data about the transmission of messages in a communications system. 

Regarding Claim 41 : 
Howard teaches, 

the processor is used to carry out access control in a communications network, [(col. 1, line 
11-16 "In the telecommunications field, large amounts of data are available, for example about 
customer behavior and telephone usage. This data contains potentially useful information for 
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many purposes such as detection of fraud, marketing, billing, maintenance planning and fault 
detection. 66 )] It would have been obvious at the time the invention was made to a person having 
ordinary skill in the art to which said subject matters pertains, to employ a processor is used to 
carry out access control in a communications network, because the processor (computer system) 
can be used to interpret, analyze and investigate user patterns of behavior in detecting e.g., 
telecommunications fraud. 

Regarding Claim 42: 
Howard teaches, 

the processor is used to carry out routing in a communications network, [(col, 1, line 11-16 "In 

the telecommunications field, large amounts of data are available, for example about customer 
behavior and telephone usage. This data contains potentially useful information for many 
purposes such as detection of fraud, marketing, billing, maintenance planning and fault 
detection. 6 ')] It would have been obvious at the time the invention was made to a person having 
ordinary skill in the art to which said subject matters pertains, to employ a processor is used to 
carry out routing in a communications network, because the processor (computer system) can be 
used to interpret, analyze and investigate user patterns of behavior in detecting e.g., in relation to 
faults in a communications network and encryption key management. 
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Conclusion 



14. Office personnel are to give claims their "broadest reasonable interpretation" in light 
of the supporting disclosure. In re Morris, 127 F.3d 1048, 1054-55, 44 USPQ2d 1023, 1027-28 
(Fed. Cir. 1997). Limitations appearing in the specification but not recited in the claim are not 
read into the claim. In re Prater, 415 F.2d 1393, 1404-05, 162 USPQ 541, 550-551(CCPA 
1969). See *also In re Zletz, 893 F.2d 319, 321-22, 13 USPQ2d 1320, 1322(Fed. Cir. 1989) 
("During patent examination the pending claims must be interpreted as broadly as their terms 
reasonably allow. . . . The reason is simply that during patent prosecution when claims can be 
amended, ambiguities should be recognized, scope and breadth of language explored, and clari- 
fication imposed. ... An essential purpose of patent examination is to fashion claims that are 
precise, clear, correct, and unambiguous. Only in this way can uncertainties of claim scope be 
removed, as much as possible, during the administrative process."), see MPEP § 2106. 
Finally, examiner interprets the "Discussion of the Related Art as prior art and applicant's 
own admission regarding what is known in the arts. 



15. Any inquires concerning this communication or earlier communications from the 
examiner should be directed to Michael B. Holmes, who may be reached Monday through 
Friday, between 8:00 a.m. and 5:00 p.m. EST. or via telephone at (571) 272-3686 or facsimile 
transmission (571) 273-3686 or email Michael.holmesb@uspto.gov . 

If attempts to reach the examiner are unsuccessful the Examiner's Supervisor, 
Anthony Knight, may be reached at (571) 272-3687. 



Correspondence Information 



Application/Control Number: 09/787,698 
Art Unit: 2121 




Michael B. Holmes 

Patent Examiner 
Artificial Intelligence 
Art Unit 2121 
United States Department of Commerce 
Patent & Trademark Office 



